UNCLASSIFIED 
AD  NUMBER 

ADC03401  6 

CLASSIFICATION  CHANGES 

TO:  unclassified 

FROM:  confidential 

LIMITATION  CHANGES 

TO: 

Approved  for  public  release,  distribution 
unlimited 


FROM: 

Distribution  limited  to  U.S.  Gov't, 
agencies  only;  Test  and  Evaluation;  6 Apr 
84.  Other  requests  must  be  referred  to  the 
Commanding  Officer,  NORDA,  NSTL  Station, 

MS  39529. 


THIS  PAGE  IS  UNCLASSIFIED 


OTIC  FILE  COPy  ad  c 0 3 4 0 1 6 


CONFIDENTIAL 


NORDA  Report  33 


The  Acoustic  Model  Evaluation  Committee 

(AMEC)  Reports 

Volume  I.  Model  Evaluation 
Methodology  and  Implementation  (U) 

Richard  B.  Lauer 

Environmental  Requirements  end  Program  Analysis  Group 
Ocnen  Science  and  Technology  Laboratory 


September  1931k 


qHgp 


DT1C 

ELECTEI 


SELEC 
APR  6 1984 

^ B 


D 


Distribution  limited  to  U.S.  Government  agencies  only. 
Other  requests  for  this  document  must  be  referred  to 
the  Commanding  Officer,  Naval  Ocean  Research  and 
Development  Activity,  NSTL  Station,  Mississippi  39529 


s apr  flea 


Classified  by  OPNAV1NST  5513.5  (3) 


Declassify  on: 

Originating  Agency's  Determination  Required 


Naval  Ocean  Research  and  Development  Activity 

NSTL  Station,  Mississippi  39529 


84  04  06  513 


■J,  Oi  v v •/  ■»_- 


CONFIDENTIAL 

(This  page  ifl  unclassified) 

Foreword  (U) 


(U)  The  Acoustic  Model  Evaluation  Committee  (AMEC)  has 
been  chartered  to  serve  as  an  advisory  group  to  the 
Director,  Naval  Oceanography  Division  (OP-952),  on 
matters  dealing  with  model  evaluation.  In  fulfillment 
of  its  charter  AMEC  will  produce  a series  of  reports 
detailing  the  results  of  model  evaluations.  This  first 
volume  describes  the  methodology  selected  and  the 
manner  in  which  it  has  been  implemented.  Subsequent 
volumes  will  present  the  results  of  specific  evalua- 
tions. Application  of  the  methodology  leads  to  infor- 
mation on  the  physics,  algorithms,  numerical  techni- 
ques and  computer-related  questions  of  use  to  program 
managers,  scientists,  system  designers,  and  fleet 
users.  Model  errors  and  deficiencies  are  given  as  are 
recommendations  for  model  improvements.  These  findings 
are  supported  by  test  cases  that  Include  comparisons 
with  data  sets  representing  a broad  range  of  environ- 
mental and  acoustic  conditions. 


G.T.  Phelps,  dpuin,  dsn 

Coflimandifti  Officir,  NORDA 
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Executive  Summary  (U) 


(U)  The  Acoustic  Model  Evaluation  Committee  (AMEC)  has 
adopted  an  interim  methodology  for  the  evaluation  of 
propagation  loss  models.  The  methodology  consists  of 
providing  the  following  information:  (a)  model  de- 

scription, (b)  physics  and  mathematics,  (c)  run  time, 
(d)  core  storage,  (e)  complexity  of  program  execution, 
( f ) ease  of  effecting  program  alterations,  (g)  program 
implementation  on  a different  computer,  (h)  cognizant 
individual (s)  or  organization  eleraent(s),  (i)  byprod- 
ucts, (j)  special  features,  and  (k)  references.  The 
accuracy  of  the  model  is  assessed  by  quantitative 
measures  of  comparison  with  referfnee  experimental 
data  sets,  other  models,  and  close,  form  solutions. 
Two  techniques  are  used:  The  difference  technique, 

whereby  differences  between  the  model  and  a reference 
are  statistically  given  in  various  regions  (e.g.,  di- 
rect path,  bottom  bounce,  convergence  zone);  and  the 
Figure  of  Merit  (FOM)  technique,  whereby  detection 
coverage  as  given  by  the  model  and  a reference  data 
set  are  compared  as  a function  of  Figure  of  Merit.  The 
basic  intent  of  model  evaluation  is  to  provide  model 
users,  sonar  system  designers,  and  those  who  select 
models  for  use  in  making  sonar  system  performance  pre- 
dictions with  basic  information  on  a model  including 
its  physical  foundations,  domains  of  applicability, 
software  configuration,  and  machine  dependencies  so 
that  the  best  match  between  acoustic  model  and  sonar 
application  may  be  achieved. 
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The  Acoustic  Model  Evaluation  Committee  (AMEC)  Reports 

Volume  I.  Model  Evaluation  Methodology  and  Implementation  (U) 

1.0  (U)  Introduction 


(U)  This  report  Is  the  first  of  a series 
presenting  the  results  of  the  Acoustic 
Model  Evaluation  Committee's  (AMEC) 
evaluation  of  specific  environmental 
acoustic  models  intended  for  operation- 
al use  or  system  design.  This  first  vol- 
ume presents  the  AMEC  charter  and  the 
approach  chosen  for  the  fulfillment  of 
that  charter.  The  methods  used  in  model 
evaluation  wirl  evolve  with  experience 
gained  and  the  ''addition  of  new  evalua- 
tion concepts  and  techniques;  the  data 
base  used  for  model  comparison  will  be 
added  to  until  it  Is  representative  In 
terns  of  acoustic  parameters  (e.g., 
frequency,  source  depth,  receiver  depth) 
and  tactical  scenarios  (e.g.,  ocean  ba- 
sin, bottom  loss  province  types,  exis- 
tence of  surface  ducts);  models  will  be 
changed  and  will  require  partLal  reeval- 
uation. Each  volume  presents  a stand- 
alone evaluation  of  a specific  model.  A 
given  model  Is  not  provided  a "seal  of 
approval";  rather,  information  is  pro- 
vided that  may  enable  a program  manager, 
project  engineer  or  research  scientist 
to  determine  if  a model  is  adequate  for 
a specific  application  and  to  compare 
various  evaluated  models  In  order  to 
select  the  optimum  model.  It  should  be 
noted  at  the  outset  that  no  single  model 
may  be  optimum  for  all  applications 
since,  for  example,  some  applications 
emphasize  speed  over  accuracy,  others 
pertain  to  a given  frequency  extent,  and 
others  require  outputs  beyond  those  or- 
dinarily available.  Specific  recommenda- 
tions for  model  improvements  are  made  in 
those  instances  where  the  improvements 
would  not  alter  the  basi t characteris- 
tics of  the  model  (e.g.,  not  substan- 
tially increase  running  time)  except  in 
cases  where  errors  have  been  found. 


(U)  Errors  in  a model  are  noted  when 
discovered  and  corrections  given.  The 
philosophy  and  approach  outlined  in  this 
volume  apply  to  propagation  loss,  ambi- 
ent noise  and  reverberation  models.  The 
specific  treatment  and  examples  are  for 
propagation  loss  models,  with  emphasis 
on  propagation  loss  models  employing 
range  independent  environmental  Inputs. 

2.0  (II)  Proposed  Charter  for  the  Acoustic  Model 
- Evaluation  Committee 

V 

(U)  Purpose:  The  purpose  of  the  Acoustic 
Model  Evaluation  Committee  (AMEC)  is  to 
ensure  that  basic  ocean  acoustic  and  re- 
lated environmental  models  are  properly 
evaluated,  so  that  the  Navy  will  have 
confidence  In  the  results  and  an  under- 
standing of  the  limitations  of  acoustic 
models  being  used  to  support  fleet  oper- 
ations and  system  design  and  analysis 
studies.^  The  AMEC  will  serve  as  an  ad- 
visory group  to  the  Director,  Naval 
Oceanography  Division  (OP-952),  on  mat- 
ters dealing  with  model  evaluation. 
Basic  acoustic  models  include  propaga- 
tion, ambient  noise,  and  reverberation 
models . 

(U)  Operations : The  AMEC  will  establish 
guidelines,  methods,  and  criteria  for 
testing  and  evaluating  candidate  models, 
and  will  ensure  that  these  guidelines, 
methods,  and  criteria  have  been  followed 
before  certifying  models  as  properly 
evaluated,  and  will  provide  guidance  for 
operational  use.  Actual  testing  and 
evaluation  is  to  be  performed  jointly  by 
AMEC  and  the  developing  organization. 

(IJ)  The  AMEC  Chairman  reports  to  the 
Director,  Naval  Oceanography  Division 
(OP-952). 
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(U)  Membership:  Members  of  the  AMEC  ire 
drawn  from  the  technical  community 
composed  of  Navy  and  university  labora- 
tory personnel  who  have  extensive  expe- 
rience in  environmental  and  acoustical 
modeling.  The  AMEC  will  meet  as  directed 
by  the  AMEC  chairman,  but  at  at  least 
three  times  annually. 

(U)  Functions:  The  AMEC  is  responsible 
for  the  following  function: 

• (U)  The  AMEC  will  establish  standard- 
ized guidelines,  methods,  and  criteria 
by  which  ocean  acoustic  and  related  en- 
vironmental models  may  be  tested  and 
evaluated . 

e (U)  The  AMEC  will  review  the  results 
of  tests  and  evaluations  of  candidate 
models  to  ensure  applicability  and  suit- 
ability for  operational  use. 

• (U)  The  AMEC  will  review  documentation 
of  evaluated  models,  including  users 
manuals,  for  completeness  and  clarity. 

• (U)  The  AMEC  will  continually  review 
the  status  of  the  ocean  acoustic  model- 
ing effort  to  determine  deficiencies, 
and  to  ensure  early  identification  of 
model  requirements  for  emerging  systems. 


of  evaluation  encompasses  ocean  environ- 
mental data  bases  and  models,  the  basic 
acoustic  building  block  models  of 
propagation  loss,  ambient  noise  and 
reverberation  levels,  system  performance 
models  and  engagement  models.  In  all 
cases,  AMEC  evaluation  is  limited  to  the 
environmental  and  acoustic  components  of 
any  model.  Initial  evaluation  efforts 
have  focused  on  propagation  loss  models 
utilizing  range  independent  environmen- 
tal Inputs.  The  evaluation  methodology 
developed  for  range  independent  propaga- 
tion loss  has  also  been  deemed  applica- 
ble for  range  dependent  propagation  loss 
models . 

(U)  In  fulfillment  of  Its  charter,  the 
AMEC  effort  consists  of  simultaneous 
tasks  in  four  major  areas:  (1)  the 

establishment  and  refinement  of  a model 
evaluation  methodology  suitable  to  each 
model  type  (e.g.,  propagation  loss  with 
range  dependent  inputs,  directional 
ambient  noise);  (2)  the  identification, 
acquisition,  and  use  of  data  sets  suit- 
able for  the  quantitative  assessment  of 
model  accuracy;  (3)  the  development  of  a 
portable  test  package  to  assure  uniform- 
ity of  model  evaluations  in  a timely, 
low  cost  manner;  and  (4)  the  performance 
of  model  evaluations. 
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• (U)  The  AMEC  will  advise  cognizant 
program  managers  relative  to  model  eval- 
uation and  model  requirements. 

• (U)  The  AMEC  will  certify  satisfactor- 
ily evaluated  models  as  Navy  Evaluated 
Models,  specifying  confidence  limits  and 
limitations,  if  any. 

• (U)  The  AMEC  will  identify  data  sets 
for  use  in  model  evaluation. 

3.0  (U)  The  Role  and  Approach  of  the  Acoustic 
Model  Evaluation  Committee 

(!J ) The  Acoustic  Model  Evaluation  Com- 
mittee’s (AMEC)  primary  function  is  the 
evaluation  of  ocean  environmental  and 
acoustic  models  Intended  for  operational 
use  and  the  design  of  systems.  The  scope 


(U)  The  evaluation  of  environmental  or 
acoustical  models  is  a joint  endeavor 
between  the  model  developer,  or  an  indi- 
vidual or  organizational  element  claim- 
ing responsibility  for  the  dissemination 
and  maintenance  of  the  model,  and  the 
AMEC  through  an  appropriate  subcommittee 
or  designee.  The  responsibilities  are 
delineated  as  follows: 

• (U)  Documentation:  The  model  developer 
provides  model  documentation  in  accord- 
ance with  requirements  as  given  in  the 
paragraph  entitled  "References"  on  page 
7.  This  response  is  reviewed  by  AMEC  for 
completeness,  clarity,  and  correctness. 
Following  discussions  between  the  AMEC 
representative  and  the  model  developer, 
the  documentation  Is  altered  and  amended 
as  necessary,  and  submitted  for  AMEC 
approval . 
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• (U)  Test  Cases:  Test  cases  selected  by 
AMEC  are  run  by  the  model  developer. 
These  test  cases  are  a subset  of  those 
available  to  AMEC,  consisting  of  experi- 
mental acoustic  data  supported  by  envi- 
ronmental datA,  environmental  acoustic 
scenarios  (the  intent  here  is  to  compare 
acoustic  results  with  those  of  an  appro- 
priate reference  model  and  diagnose  the 
causes  of  significant  di screpancie si , 
and  closed  form  solutions.  Further  test 
cases  are  based  upon  an  examination  of 
the  physics  and  mathematics  of  the  model 
by  an  Independent  expert  (see  below). 
The  test  cases  are  meant  to  reflect  the 
model's  intended  applications  and  stated 
domains  of  applicability  and  Limita- 
tions. They  are  to  be  produced  in  a 
prescribed  format  with  other  supporting 
information  (e.g.,  run  time).  The  envi- 
ronmental inputs  are  supplied  by  AMEC. 
If  input  or  output  requirements  cannot 
be  met  by  the  model  to  be  evaluated, 
agreed  upon  alterations  may  be  made 
following  discussions  between  the  model 
developer  and  AMEC.  (Note:  such  problems 
and  alterations  are  to  be  fully  documen- 
ted and  included  in  the  final  report.) 

e (U)  Accuracy  Assessment:  The  results 
of  test  cases  are  compared ' to  experimen- 
tal data,  results  of  a reference  model, 
or  closed  form  solutions,  as  appropri- 
ate. The  comparisons  are  quantitative 
and  are  performed  under  AMEC  control 
using  approved  AMEC  techniques.  (Note: 
The  accuracy  assessment  techniques  and 
test  cases  are  evolutionary;  therefore, 
refinements  and  additions  are  expected 
with  the  passage  of  time.) 

• Physics  and  Mathematics  Examina- 
tion: An  exam l nation  of  the  model 1 s 

foundations  in  terns  of  its  physics  and 
mathematics  are  undertaken  by  an  expert 
selected  by  AMEC.  Particular  attention 
is  given  to  aspects  of  the  formulation, 
especially  assumptions  and  approxima- 
tions that  may  tend  to  limit  the  model's 
domain  of  applicability.  Such  limita- 
tions should  be  tested  by  the  identifi- 
cation of  appropriate  test  cases,  the 
running  of  these  cases  by  the  model 
developer  and,  finally,  the  comparison 


of  the  results  with  those  of  a reference 
model,  if  available. 

4.0  (U)  The  Mechanics  of  Model  Evaluation:  A 
Multi-State  Process  Requiring  a Portable 
Test  Package 

(U)  The  portable  test  package  is  needed 
for  the  simultaneous  evaluation  of 
several  models.  The  evaluations  are  to 
be  carried  out  expeditiously  while  re- 
taining AMEC  control  of  certain  crucial 
steps,  particularly  those  involving  com- 
parison of  model  results  with  represen- 
tative data  sets  retained  by  AMEC  for 
the  purpose  of  accuracy  assessment.  It 
is  also  essential  that  all  analyses 
functions  incidental  to  model  evaluation 
be  performed  under  supervision  of,  and 
review  by,  AMEC.  Efficiency  and  control 
are  achieved  by  a two-stage  model  evalu- 
ation. Stage  I is  fulfilled  by  the  model 
developer  or  persons  at  the  developing 
activity;  Stage  II  is  carried  out  by 
designees  of  AMEC. 

(U)  In  Stage  I,  the  model  developer  pro- 
vides user-oriented  information  by  means 
of  filling  out  a model  information  form. 
The  areas  covered  include:  (1)  model 

description  comprising  the  model's  pur- 
pose, input/output  information,  systems 
presently  using  or  intending  to  use  the 
model;  (2)  core  storage  requirements 
(including  assumed  number  of  bits  per 
word);  (3)  a flow  chart;  (4)  a program 
listing;  (5)  a list  of  computers  on 
which  the  program  is  running;  (6)  spe- 
cial requirements  such  as  computer 
language,  special  codes  (e.g.,  for  plot 
generation),  word  length,  and  library 
routines;  (7)  a list  of  any  versions  of 
the  program  extant  which  differ  from 
that  undergoing  evaluation  and  the  exact 
nature  of  the  differences;  (8)  defini- 
tion of  all  parameters  used  in  running 
the  program  including  default  values  and 
guidance  for  the  selection  of  any  unusu- 
al parameters;  (9)  the  name(s)  of  cogni- 
zant individual( s)  or  organizational 
elements  who  formulated  the  program, 
were  responsible  for  its  computer  imple- 
mentation and  are  now  responsible  for 
program  maintenance;  (10)  references. 
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including  user's  guides,  response  to 
3ECNAVIN3T  3560.4,  and  specific  algo- 
rithms used;  (11)  additional  information 
produced  In  addition  to  the  main  func- 
tion or  as  a byproduct;  and  (12)  special 
features  (e.g.,  ability  to  impose  source 
and  receiver  beam  patterns  In  propaga- 
tion loss  models  or  availability  of  a 
frequency  interpolation  scheme).  Also  in 
stage  1,  the  model  developer  runs  the 
model  for  test  cases  specified  by  AMEC. 
Inputs  to  the  model  and  output  format 
are  specified  by  AMEC.  If  for  some 
reason  the  model  cannot  utilize  the  in- 
puts in  the  form  given  or  cannot  produce 
the  required  output,  mutually  agreeable 
alterations  will  be  found  and  document- 
ed, including  the  reason(s)  for  the  in- 
ability to  follow  the  original  request. 
The  test  cases  will  be  designed  to  eval- 
uate the  model's  accuracy  over  a broad 
range  of  representative  environmental 
scenarios.  Test  results  are  to  be  re- 
turned to  AMEC  In  the  specified  format 
In  addition  to  run  decks  for  a specified 
subset  of  cases  and  running  time  for  all 
cases • 

(U)  Stage  II  of  model  evaluation  con- 
sists of  an  AMEC  designee  compiling  the 
results  from  stage  I.  This  compilation 
is  followed  by  analysis  of  the  model 
Information  form  results  for  complete- 
ness and  clarity.  Areas  identified  as 
Incomplete,  unclear  or  of  questionable 
validity  are  resolved  In  a second  Itera- 
tion with  the  model  developer.  Results 
of  the  test  cases  are  compared  with  data 
(experimental,  closed-form  solutions  and 
results  of  "reference"  models)  compiled 
by  AMEC.  The  result  of  this  process  Is  a 
quantitative  assessment  of  model  accura- 
cy over  representative  scenarios.  Where 
possible,  discrepancies  are  analyzed, 
model  deficiencies  or  limitations  iden- 
tified and  recommendations  for  model  im- 
provement given.  The  quantitative  accu- 
racy assessment  of  Stage  II  Is  perfoi*med 
by  means  of  AMEC-approved  techniques 
that  have  been  computer  Implemented. 


(U)  For  propagation  loss  model  evalua- 
tion, AMEC  has  adopted,  on  an  Interim 
basis,  the  evaluation  methodology  devel- 
oped by  the  Panel  on  Sonar  System  Models 
(POSSM)  as  given  In  Lauer  and  Sussman 
(1976,  1979).  The  quantitative  accuracy 
assessment  algorithms  and  associated 
graphics  are  contained  In  a computer 
program  known  as  MCPRO  (Sussman  and 
Oberlander,  1979).  This  program,  origi- 
nally developed  at  the  New  London  Labo- 
ratory, Naval  Underwater  Systems  Center, 
has  been  disseminated  to  the  Naval  Ocean 
Systems  Center  and  the  Naval  Ocean 
Research  and  Development  Activity.  Data 
tapes  for  use  in  accuracy  assessment  are 
to  be  generated  for  each  model  type 
(i.e.,  range  Independent  propagation 
los 8 (R1PL),  range  dependent  propagation 
loss  (RDPL),  directional  ambient  noise 
(AN)  and  reverberation  level  (RL)).  To- 
gether, these  tapes  will  likewise  be 
disseminated.  The  program  MCPRO  and  the 
tape  of  data  sets  for  range  independent 
propagation  loss  model  accuracy  assess- 
ment will  constitute  a portable  test 
package  for  use  In  stage  II  of  the  eval- 
uation process. 

S.O  (U)  Elements  of  Model  Evaluation 

(U)  Each  model  evaluation  (to  be  report- 
ed in  subsequent  volumes  of  this  scries) 
consists  of  a number  of  elements  that 
will  provide  information  which  should  be 
of  assistance  In  selecting  a model  for  a 
given  application.  These  elements  are: 

• Model  Description 

• Physics  and  Mathematics 

• Run  Time 

• Core  Storage 

• Complexity  of  Program  Execution 

• Ease  of  Effecting  Program 
Alterations 

• Ease  of  Implementation  (On  a 
Different  Computer) 

• Cognizant  Individual  or 
Organization  Elements 

• References 

• Byproducts 

• Special  Features 

• Accuracy  Assessment  Quantitative) 
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(U)  in  the  following  sections,  each 
evaluation  element  is  defined  in  detail, 
it  is  not  expected  that,  in  a specific 
evaluation,  this  level  of  detail  will  be 
achieved  for  many  elements.  The  ele- 
ments, as  defined  below,  may  therefore 
be  thought  of  as  a checklist.  Many  of 
the  specific  Items  in  the  checklist 
would  normally  entail  expenditures  of 
time  and  money  above  and  beyond  those 
available  for  model  evaluation  efforts, 
and  are  included  in  the  hope  that  they 
may  be  available  and  also  for  the  sake 
of  completeness. 

(U)  The  evaluation  elements  h^lp  define 
a model's  domain  of  applicability  and 
Its  limitations.  Where  applicable,  rec- 
ommendations will  be  made  regarding 
model  usage  and  Improvements.  A detail- 
ed description  of  the  evaluation  ele- 
ments follows. 

5.1  (U)  Model  Descriptiion 

(U)  The  model  description  provides  in- 
formation in  a variety  of  areas:  (1)  The 
purpose(s)  of  the  model  is  set  forth  and 
is  punctuated  by  providing  examples  of 
the  model's  outputs,  including  tahular 
and  graphic  results,  as  applicable.  The 
extent  to  which  outputs  are  under  user 
control  is  identified.  All  output  op- 
tions are  catalogued.  (2)  A list  of 
I lput  variables  and  their  units.  Inputs 
obtained  from  associated  data  bases  or 
from  internal  routines  or  tables  should 
be  identified.  This  listing  Is  to  be 
functional  and  does  not  include  a review 
of  the  physics  upon  which  any  input 
algorithms  are  based,  such  reviews  being 
four*!  In  the  "physics  review"  section. 
(3)  A list  of  systems  supported  by  the 
model.  This  list  Is  to  Include  the  rr  le 
performed  by  the  model  In  the  system  and 
the  stated  purpose  of  the  system.  (4) 
Any  limitations  designed  into  the  model 
through  Inherent  limits  of  the  physics, 
mathematics,  environmental  description, 
computer  implementation,  etc.  These 
limitations,  taken  together,  define  the 
model's  domain  of  applicability.  Limi- 
tations are  given  In  terms  of  parameters 
such  as  frequency,  bandwidth,  range, 


depth,  angle,  and  time.  Also  Included 
are  limitations  involving  computer  gra- 
phics or  telemetry  links.  These  limita- 
tions usually  result  from  design  deci- 
sions based  upon  the  basic  purpose  of 
the  model  development  effort  i r trade- 
offs required  by  time,  cost,  and  compu- 
ter assets.  (5)  A list  of  extant  model 
versions.  These  would  Include  versions 
for  research  use  and  those  adapted  for 
system  usage,  land-based  and  shipboard 
versions,  and  those  adapted  for  special 
computer  requirements.  Changes  in  Input 
and  output  are  noted,  as  are  default 
values  of  various  parameters. 

5.2  (U)  Physics  and  Mathematics 

(U)  The  model  Is  examined  in  terms  of 
Its  physical  and  mathematical  basis.  The 
examination  is  performed  by  an  indepen- 
dent expert  In  the  appropriate  field  of 
modeling.  In  particular,  the  physics  and 
mathematics  are  examined  to  define  the 
model's  domain  of  applicability  through 
assumptions,  approximations  and  the  as- 
signment of  "nominal  values"  to  various 
parameters.  For  propagation  loss  model- 
ing, examples  of  the  above  are: 

• (TJ)  Assumption : Horizontally  strati- 
fied raedi  im,  rendering  the  model  incapa- 
ble of  correctly  predicting  propagation 
across  thermal  fronts  or  in  areas  of 
varying  bathymetry. 

• (U)  Approximation:  The  merging  of  deep 
portions  of  historical  temperature  rec- 
ords to  bathythermograph  data,  possibly 
causing  errors  in  levels  of  the  ranges 
at  which  features  appear  or  even  the 
prediction  of  features  which  do  not 
exist  In  reality  under  certain  environ- 
mental conditions. 

• (U)  Nominal  values:  In  some  models, 

rays  radiating  from  the  source  up  to  a 
specified  angle  are  accounted  for  In  the 
propagation  loss  computations.  The  maxi- 
mum value  of  this  angle  may  be  under 
user  control  or  assigned  a nominal  value 
based  on  environmental  expectations  and 
running  time  requirements.  In  environ- 
ments which  exceed  the  expectation, 
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implied  by  the  assigned  nominal  value, 
significant  contributions  to  the  total 
received  level  would  be  neglected. 

(U)  The  reporting  of  the  model's  physics 
includes  the  basic  foundations  and  ap- 
proach and  any  unusual  techniques  and, 
particularly,  any  extensions  to  theory 
or  unique  capabilities  otherwise  un- 
available. Examination  of  the  model's 
physics  and  mathematics  is  to  include 
consideration  of  environmental  Inputs, 
including  theories  and  their  implementa- 
tion and  appropriateness  of  environmen- 
tal data  base  selection. 

(U)  It  is  assumed  that  mathematical  no- 
tation will  be  employed.  However,  corre- 
sixjndeoce  between  mathematical  symbology 
and  variable  names  used  by  the  program 
should  he  assured  by  a correspondence 
list . 

5.3  (U)  Run  Time 

(U)  Run  time  of  a model  is  a function  of 
computer,  number  of  points  per  predic- 
tion and  input/output  selections.  Run 
times  are  given  for  a number  of  environ- 
mental/acoustic scenarios  with  appropri- 
ate identification  of  the  above  factors. 
Information  on  run  time  for  each  version 
of  a model  is  desirable.  Run  time  is  to 
be  divided  into  time  required  for  compu- 
tation and  time  for  peripheral  require- 
ments such  as  plotting.  Any  tradeoffs 
between  run  time  and  accuracy  should  be 
described,  including  any  relation  to  en- 
vironmental and  acoustic  inputs. 

5.4  (U)  Core  Storage 

(U)  Core  storage  wilL  vary  with  model 
version,  perhaps  driven  by  available 
core  and  word  length.  Any  efforts  to 
reduce  core  storage  requirements  are  to 
be  identified  such  as  overlays  (Indepen- 
dent subroutines  being  brought  into 
addressable  memory  as  needed  and  removed 
upon  execution),  memory  mapping  (addres- 
sing normally  nonaddressable  core  by 
executive  calls),  disk  memory  swap  (data 
brought  from  disk  into  memory  a small 
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amount  at  a time,  calculations  perform 
ed , results  returned  to  disk,  and  the 
next  data  portion  brought  into  core, 
etc.)  and  use  of  techniques  such  as  in- 
terpolations in  place  of  calculation. 

5.5  (U)  Complexity  of  Program  Execution 

(U)  Complexity  of  program  execution  is 
directly  related  to  the  options  avail- 
able to  a user.  In  a model  to  be  used 
for  research  applications,  complex  and 
varied  inputs  may  be  required,  special 
outputs  at  great  levels  of  detail  for 
diagnostic  usage  may  be  incorporated, 
and  accuracy  requirements  ace  likely  to 
be  extremely  stringent.  Thus,  for  re- 
search-oriented models,  input  and  output 
options  must  be  broad  with  parameters 
affecting  accuracy  under  user  control 
implying  great  complexity  of  program  ex- 
ecution. In  contrast,  a model  Intended 
for  system  usage  or  support,  especially 
a shipboard  application,  should  assign 
as  many  parameters  as  possible  internal- 
ly and  be  simple  to  execute.  The  simpli- 
city of  execution  Is  achieved  at  the 
expense  of  an  accurate,  precise  environ- 
mental description  (which  may  be  due  to 
insufficient  environmental  sensing 
ability)  or  decreased  accuracy  of  compu- 
tations (often  affordable  in  a broad 
system  context)  . 

(U)  Complexity  of  program  execution  is 
revealed  by  providing  (1)  a sample  pro- 
gram listing,  (2)  definition  of  all  in- 
put and  output  parameters  under  user 
control,  (3)  default  values  or  condi- 
tions assigned  within  the  program,  (4) 
restrictions  on  the  values  of  arguments, 
(5)  a detailed  explanation  of  unusual 
parameters  with  guidance  for  their  se- 
lection, and  (h)  reference  to  a user's 
guide . 

5.6  (U)  Ease  of  Effecting  Program  Alterations 

(U)  It  is  not  uncommon  that  a program 
will  satisfy  most  requirements  for  a 
given  application  but  require  some  mod- 
ifications before  all  criteria  can  be 
met.  Modifications  may  range  from 
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changing  U.S.  customary  to  metric  units 
to  altering  the  program  to  meet  core 
storage  or  running  time  requirements. 
Other  alterations  include  the  assignment 
of  default  values  to  certain  variables 
in  going  from  research  to  system  up-ge, 
the  inclusion  of  archival  data  ba..e 
and  elimination  of  routines  to  calculate 
values  of  some  needed  variables  that  may 
be  obtainable  in  situ,  such  changes  typ- 
ical of  land-based  models  being  adapted 
for  shipboard  use.  Other  significant 
alterations  involve  the  forms  of  printed 
and  graphic  outputs,  and  most  important- 
ly, in  many  cases  interfaces  between  the 
model  and  other  models  of  the  system. 
The  ease  of  effecting  program  altera- 
tions is  difficult  to  assess  a priori. 
Of  course,  the  more  modularized  a 
program,  the  easier  it  is  to  change  one 
portion  without  affecting  others.  Hence, 
information  on  program  architecture  and 
flow  can  provide  valuable  clues  as  Lo 
the  difficulty  of  effecting  alterations. 
In  addition  to  a flow  chart,  a program 
listing  provides  the  most  basic  informa- 
tion, particularly  when  supported  by 
documentation — both  external  to  the 
program  and  internal  in  the  form  of 
comment  cards.  Lists  of  variables  and 
their  definitions  are  critical;  choice 
of  variable  names  consistent  with  common 
usage  is  very  helpful  in  following  a 
program's  coding.  Other  information  of 
extreme  importance  in  effecting  program 
alterations  Is  the  extent  to  which  a 
program  is  tied  into  a computer  execu- 
tive systan  or  other  special  equipment 
or  programs. 


language,  word  length  of  the  machines  on 
which  the  program  is  presently  running, 
special  codes  such  as  those  necessary 
for  plotting  routines,  and  specialized 
library  functions.  Any  tie-in  to  the  ex- 
ecutive system  of  a given  computer  is 
especially  pertinent  if  implementation 
on  another  computer  is  to  be  performed. 

(U)  To  assure  correctness  of  implemen- 
tation on  a given  computer,  test  cases 
that  exercise  all  subroutines  are  neces- 
sary. A limited  number  of  test  cases 
would  indicate  correctness  of  implemen- 
tation. All  error  returns  should  be 
documented  as  to  location  in  the  program 
and  the  nature  of  error  indication. 

5.8  (U)  Cognizant  Individual(s)  and/or 
Organizational  Element(s) 

(U)  In  any  specific  application  of  a 
model,  questions  inevitably  arise  that 
are  not  answerable  by  reference  to 
available  documentation.  Such  questions 
often  require  lengthy  discussions  with 
those  who  have  developed  the  model  or 
have  been  responsible  for  its  computer 
implementation  and,  in  some  cases,  those 
who  have  developed  the  theory  upon  which 
the  model  is  based.  In  some  cases  an 
organizational  element  is  responsible 
for  model  configuration  and  maintenance. 
In  any  intended  alteration  of  the  model 
or  desire  to  implement  the  model  on  a 
new  computer,  one  of  the  aforementioned 
sources  should  be  consulted. 

5.9  (U)  References 


5.7  (U)  Ease  of  Implementation  (On  a 
Different  Computer) 

(U)  Given  that  a program  is  running  on 
one  computer,  we  art  often  confronted 
with  assessing  the  difficulty  with  which 
the  program  can  be  adapted  to  another 
computer.  The  most  basic  information 
helpful  in  this  regard  is  a list  of  com- 
puters on  which  the  program  is  running. 
Of  course,  if  the  come  ter  of  interest 
is  on  this  list,  the  problem  is  likely 
solved.  If  not,  other  information  can  be 
extremely  useful,  such  as  the  computer 


(U)  A list  of  references  provides  valu- 
able source  material  for  detailed  infor- 
mation about  various  aspects  of  model 
theory,  implementation  and  usage.  Refer- 
ences worthy  of  special  mention  are:  (l) 
a user's  guide;  (2)  a response  to  SECNA- 
VINST  3560.1,  Tactical  Digital  Systems 
Documentation  Standards,  8 August  1974; 
and  (3)  references  to  theories  and  nu- 
merical techniques  employed  by  the 
model . 
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5.10  (U)  By-Products 

(U)  Models  often  are  capable  of  produ- 
cing useful  byproducts  at  little  or  no 
extra  cost  in  running  time  or  core 
storage.  These  byproducts  may  serve  as 
diagnostic  tools  or  may  be  valuable  in 
their  own  right  as  exemplified  by  a ray 
diagram  associated  with  a propagation 
loss  model.  Often,  an  output  byproduct 
is  an  essential  input  for  another  model, 
such  as  t**e  travel  time  of  ray  paths  as 
a function  of  range  being  a byproduct  of 
a propagation  loss  model  (a  required 
input  for  reverberation  modeling.) 

(U)  Tne  production  of  byproducts  by  a 
model  may  result  in  a single  program 
taking  the  place  of  two  or  more  pro- 
grams. Often  the  production  of  byprod- 
ucts exists  in  the  form  of  exercisable 
options  with  minimal  impact  on  the  other 
factors  influencing  model  selection. 

5.11  (U)  Special  Features 

(U)  To  achieve  closer  consonance  with 
system  features  or  for  the  sake  of 
economics  in  running  time  or  core  stor- 
age, special  features  are  often  incorpo- 
rated into  a model.  For  example,  a prop- 
agation loss  model  may  have  provisions 
for  beam  patterns  to  be  imposed  on  a 
source  and  receiver  consistent  with  a 
target  radiation  characteristic  and  own 
ship's  sonar.  Using  interpolation  in 
predicting  results  at  several  frequen- 
cies can  substantially  reduce  running 
time  below  that  required  for  calculation 
of  results  at  each  frequency. 

5.12  (U)  Accuracy  Assessment  ( Quantitative ) 

(U)  The  quantitative  accuracy  assess- 
ment procedures  compare  the  data  genera- 
ted by  a model  with  a reference  data 
set,  be  it  measured  data,  the  output  of 
another  model  or  data  representing  an 
exact  solution.  As  shall  be  seen  below 
the  data  are  examined  by  two  methods, 
the  first  insofar  as  possible  Isolates 
propagation  modes  (i.e.,  direct  path, 
bottom  Interaction  and  convergence 


zone)  identifying  the  range  interval 
over  which  each  given  mode  dominates, 
and  then  comparing  the  two  data  sets  In 
each  interval  by  generating  statistics 
of  their  differences.  One  advantage  of 
this  procedure  is  that  the  range  inter- 
vals are  those  of  tactical  interest; 
therefore,  the  .success  of  a model  in 
each  of  these  regions  Is  a useful  way  to 
delineate  model  performance.  The  second 
method  utilizes  figure  of  merit  to  com- 
pute detection  ranges  and  to  compare 
them.  It  is  obvious  that  thus  far  we 
have  restricted  our  attention  to  propa- 
gation loss  models  and  It  Is  only  for 
this  model  type  that  quantitative  accu- 
racy assessment  procedures  have  been  de- 
fined. It  1 s not  difficult,  however,  to 
see  parallels  between  propagation  loss, 
ambient  noise  and  reverberation.  The 
Independent  variables  of  significance 
are  range,  angle,  and  time,  respective- 
ly. Whereas  propagation  loss  Is  deline- 
ated into  modes  of  direct  path,  bottom 
Interaction  and  convergence  zone,  ambi- 
ent noise  is  differentiated  into  ship- 
and  wind-generated  noise  and  reverbera- 
tion may  be  divided  into  surface,  volume 
and  bottom  contributions.  Although  ad- 
mittedly the  analogies  do  not  represent 
one-to-one  correspondence,  they  should 
be  of  use  In  constructing  quantitative 
accuracy  assessment  procedures  for  ambi- 
ent noise  and  reverberation  models. 

(IJ)  The  detailed  accuracy  assessment 
procedures  for  propagation  loss  models 
are  as  follows: 

• (U)  Step  (1).  Smooth  the  reference 

data  set  and  the  output  of  the  model  to 
be  evaluated  according  to  the  following 
rules:  smooth  measured  data  If  CW  but 

not  if  1/3-octave  such  as  obtained  from 
explosive  sources;  smooth  model  outputs 
of  coherent  phase  addition  were  used  but 
not  If  incoherent  or  semi-coherent  phase 
addition  was  used.  The  purpose  of  this 
step  is  to  remove  rapid  fluctuations, 
thereby  putting  data  into  a form  where 
mean  levels  may  be  compared.  This  is  a 
crucial  point  and  bears  further  explana- 
tion. In  a sense  there  are  two  sonar 
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equations,  one  for  mean  levels  and  one 
for  the  statistically  fluctuating  com- 
ponent. The  sonar  equation  for  mean 
levels  is  the  one  commonly  referred  to 
as  "the  sonar  equation."  An  example  of 
the  other  sonar  equation  is  that  the 
standard  deviation  of  the  signal  excess 
is  equal  to  the  square  root  of  the  sura 
of  the  squares  of  the  standard  devia- 
tions of  the  factors  of  the  mean  level 
sonar  equation.  This  particular  form 
assumes  Independent  Gaussian  variables; 
other  forms  can  be  constructed  that  do 
not  contain  such  restrictive  assump- 
tions. The  point  here  is  the  necessity 
of  breaking  up  the  propagation  loss  into 
the  two  components  and  comparing  the 
model  output  with  a reference  data  set 
on  the  basis  of  those  components.  In  the 
AMEC  procedure  only  the  mean  levels  are 
compared.  They  are  obtained  by  means  of 
data  smoothing,  specifically  by  the 
application  of  a 2 kilometer  running 
average.  The  window  size  of  2 kilometers 
Is  consistent  with  sonar  system  integra- 
tion times  and  typical  target  speeds 
(e.g.,  a target  closing  at  approximately 
12  knots  and  being  detected  on  a sonar 
system  with  a five-minute  Integration 
time  or  a target  traveling  at  a relative 
bearing  of  60  degrees  with  respect  to 
own  ship  3 1 a speed  of  24  knots,  once 
again  being  detected  on  a system  with 
five-rainute  integration  time).  Signifi- 
cantly larger  averaging  intervals  smear 
some  of  the  significant  propagation  loss 
features  vAiile  smaller  averaging  inter- 
vals lead  to  the  rapid  fluctuations 
dominating  the  propagation  loss  field, 
and  the  desired  mean  level  comparison  is 
lost . 

(U)  Figure  1 gives  a measured  set  of 
1/3-octave  data,  and  Figure  2 the  output 
of  a model  that  used  coherent  phase  ad- 
dition, and  Figure  3,  a smoothed  version 
of  the  coherent  model  output  obtained  by 
performing  running  average  with  a 2 kil- 
ometer window. 

• (U)  Step  (2).  Subtract  the  model  out- 
put from  the  reference  data  set  (after 
both  have  been  appropriately  smoothed). 
Figure  4 Is  the  result  of  such  a 
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(U)  Figure  1.  PARKA  Data:  Source  Depth-50  ft, 
Receiver-Depth  300  ft,  Frequency-400  Hz 
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(U)  Figure  2.  RAYMODE  X (Coherent):  Source  Depth-50  ft, 
Receiver  Depth-300  ft,  Frequency-400  Hz 
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(U)  Figure  3.  RAYMODE  X (Coherent):  Source 
Depth-50  ft,  Receiver  Depth  300  ft,  Frequency-400  Hz 
Sliding  Averages  of  5 Points  (2.0  km) 
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subtraction,  whereby  the  smoothed  model 
ouput  of  Figure  3 was  subtracted  from 
the  measured  data  set  of  Figure  1.  After 
this  step,  the  model  under  evaluation 
and  the  reference  data  set  are  insepar- 
able. The  aim  here  is  not  to  set  up  the 
reference  as  representing  absolute  truth 
and,  hence,  all  differences  Interpreted 
as  errors  in  the  model  under  evaluation. 
The  aim,  rather,  Is  to  observe  problem 
areas  and  try  to  diagnose  the  cause(s) 
of  large  disparities,  be  they  due  to 
model  under  evaluation  or  reference 
(barring  the  case  of  closed  form  solu- 
tions). The  difference  curve  and  prod- 
ucts derived  therefrom  must,  be  interpre- 
ted with  great  care,  least  erroneous 
conclusions  be  drawn,  a point  which  will 
be  addressed  in  Step  (6)  and  is  support- 
ed by  an  example. 

• (U)  Step  (3).  Divide  the  difference 
curve  into  physically  significant  range 
intervals,  if  possible  correspondirg  to 
direct  path,  bottom  interaction  and  con- 
vergence zone  modes  of  propagation. 
These  range  intervals  are  determined 
from  the  reference  data  set  (possibly 
with  the  help  of  ray  diagram).  Figure  5 
presents  range  intervals  chosen  from  the 
reference  data  set.  In  the  event  that 
clearly  identifiable  features  associated 
with  propagation  mode  are  not  available 
In  the  reference  data  set,  three  choices 
are  available:  (a)  do  not  divide  the  da- 
ta into  range  intervals;  ( b)  divide  the 
data  into  quasl-arbitrary  intervals  that 
may  be  tactically  useful  (e.g.,  5 km  in- 
tervals out  of  20  km,  20  km  intervals 
out  of  100  km,  50  km  Intervals  thereaf- 
ter); and  (c)  when  clear  features  are 
available  to  which  it  is  difficult  to 
assign  a single  simple  physical  mech- 
anism; nonetheless,  use  these  features 
as  the  basis  of  forming  range  intervals. 

• (H)  Step  (4).  In  each  range  interval 
calculate  the  mean  /i  and  standard  devia- 
tion a of  the  differences.  Table  1 gives 
M's  and  <t's  calculated  from  the  curve  of 
Figure  4 in  the  Intervals  of  Figure  5. 
The  need  for  both  a mean  value  and  a 
measure  of  spread  (a)  is  obvious  upon 
considering  the  results  for  the  region 


UNCLASSIFIED 

(U)  Figure  4.  Smoothed  RAYMOOE  X (Coherent):  Source 
Depth-50  ft,  Receiver  Depth-300  ft,  Frequency-400  Hz 
Subtracted  from  Experimental  Data,  Source  Depth- 
50  ft,  Receiver  Depth-300  ft,  Frequency-400  Hz 
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(U)  Figure  5.  PARKA  Data:  Source  Depth-50  ft, 
Receiver  Depth-300  ft,  Frequency-400Hz 
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(U)  Table  1.  Means  and  standard  deviations  of 
differences  between  PARKA  and  smoothed 
RAYMODE  X (Coherent)  results  in  dB 
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"Beyond  Second  Convergence  Zone"  where 
observation  of  only  the  mean  valve  would 
lead  to  a conclusion  of  excellent  agree- 
ment which  consideration  of  the  value 
for  a quickly  dispels* 

• (U)  Step  (5).  Characterize  convergence 
zones  by  giving  range  of  onset,  range 
extent,  minimum  transmission  loss  In 
decibels  and  shape  (i.e.,  double  vs. 
single  lobe).  The  range  of  zone  onset  is 
given  in  terms  of  a selected  transmis- 
sion loss  level  In  decibels  as  Is  the 
range  extent  of  the  zone . 

• (U)  Step  (6).  Analyze  the  results  of 
Steps  1-5,  attempting  to  diagnose  the 
cause(s)  of  serious  discrepancies  be- 
tween the  model  under  evaluation  and  the 
reference  data  set.  This  step  Is  essen- 
tial, since  the  means  and  standard  devi- 
ations can  be  quite  misleading  as  the 
following  example  will  Illustrate.  Con- 
sider the  two  propagation  loss  curves, 
representing  convergence  zones,  of  Fig- 
ure 6a,  one  representing  the  model  under 
evaluation,  the  other  representing  the 
reference.  As  can  be  seen  the  two  curves 
are  Identical  except  for  a displacement 
in  range.  Upon  taking  differences  be- 
tween the  curves,  as  required  by  Step  2, 


(U)  Figure  6.  Comparison  of  Propagation  Loss  vs. 
Range  for  Reference  and  Model  Over  Interval  (Rj,  R2) 


Figure  6b  is  obtained.  Assuming  the 
range  interval  (Rl,  R2)  has  been  that 
chosen  from  Step  3,  we  then  would  pro- 
ceed to  calculate  the  mean  n and  stand- 
ard deviation  <r  • The  result  would  be  a 
mean  value  of  zero  and  a very  large 
standard  deviation.  Of  course,  an  ap- 
propriate translation  of  one  of  the 
curves  of  Figure  6a  would  result  In 
r«-0.  We  see  therefore  that  the  large 
standard  deviation  can  be  somewhat  mis- 
leading but  is,  in  this  example,  a "red 
flag"  inviting  us  to  determine  the  cause 
of  translation,  such  as  the  model  lack- 
ing a curved  earth  correction  or  an  er- 
roneous value  for  the  sound  speed  gradi- 
ent in  the  deep  portion  of  the  profile. 
The  essential  lessons  of  the  example  are 
that  the  results  of  a single  step  such 
as  Step  4 should  not  be  viewed  alone, 
but  rather  as  a member  of  a hierarchy  of 
results  that  begin  with  the  basic  curves 
and  that  Step  6 analysis  is  of  paramount 
importance.  We  note  in  passing  that,  in 
the  example  given.  Step  5 would  have  de- 
fined the  problem  giving  a difference  in 
convergence  zone  onset  range  for  the  two 
curves,  but  identical  results  for  zone 
duration  and  peak  level. 

(U)  The  second  quantitative  method  of 
accuracy  assessment  consists  of  present- 
ing a table  of  detection  range  as  a 
function  of  figure  of  merit.  As  in  the 
earlier  method,  we  apply  the  technique 
to  smooth  results  and  therefore  begin 
with  Step  1 which  is  to:  smooth  mearured 
data  if  CW  but  not  if  1/3-octave,  such 
as  obtained  from  explosive  sources,  and 
to  smooth  model  output  ff  coherent  phase 
addition  was  used  but  not  if  incoherent 
or  semi-coherent  phase  addition  was 
used.  The  next  procedure  is  to  select 
figure  of  merit  (FOM)  values  (for 
volumes  II  and  III  of  this  series  the 
procedure  was  not  automated,  and  5 dB 
intervals  were  selected;  for  future 
propagation  loss  model  evaluations,  an 
automated  procedure  is  available  as  de- 
scribed in  Brunson  (in  prep.)  for  which 
tabular  and  graphical  output  is  avail- 
able with  FOM  intervals  selectable  to 
1 dB  intervals).  Upon  selection  of  FOM 
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values,  the  corresponding  detection 
ranges  (multiple  range  Intervals  for  a 
single  FOM  value)  are  found  as  Indicated 
In  Figures  7 and  8 (which  are  the  data 
sets  of  Figs.  1 and  3).  As  can  be  seen 
In  Figure  7,  linear  Interpolation  is 
used  when  data  points  are  widely  spaced. 
(Note:  For  a given  FOM,  for  example, 
F0M-90  dB,  we  see  that  In  Fig.  7 the 
transmission  loss  (TL)  Is  always  less 
than  90  dB  until  a range  of  36.5  km.  The 
TL  curve  remains  below  90  dB  until  the 
first  convergence  zone.  Between  55.5  and 
59.0  km,  the  TL  is  greater  than  90  dB. 


100  110 120  130  M0  ISO  iso  TTO  iso  190  20 

RANGE  I KM  I 

UNCLASSIFIED 

(U)  Figure  7.  Experimental  Data:  Source  Depth-50  ft  , 
Receiver  Depth-300  ft,  Frequency-400  Hz 
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170  1 80  190 
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(U)  Figure  8.  RAYMODE  X (Coherent):  Bottom  Lo*s-MGS6, 
Frequency-400  Hz,-  Sliding  Average*  of  5 Point*  (2.0  km) 


This  occurs  again  in  the  second  conver- 
gence zone  between  114.5  and  120.0  km.) 
These  results  are  finally  tabulated  as 
exemplified  by  Table  2.  The  contents  of 
the  table  are  not  purely  numeric  and 
verbal  descriptions  are  used  when  war- 
ranted. (Note:  When  a TL  curve  oscil- 
lates about  a given  FOM  over  some  stated 
range  interval,  the  concept  of  zonal 
detection  coverage  (ZDC)  is  introduced, 
ZDC  is  defined  to  be  the  percentage  of 
the  range  interval  over  which  the  FOM  is 
less  than  the  TL.) 

6.0  (U)  Data  far  Accuracy  Assessment  - An 
Environmental/Acoustic  Matrix 

(U)  The  as >essment  of  model  accuracy  is 
obtained  by  comparing  the  model's  output 
with  three  basic  data  types:  (1)  exper- 

imental acoustic  results,  (2)  output  of 
a reference  model  and  (3)  closed  form 
solutions . 

(U)  The  experimental  acoustic  data  is 
supported  by  environmental  data  suffi- 
cient for  model  input  requirements. 
Model  results  are  compared  with  experi- 
mental acoustic  data  using  standard 
comparison  criteria.  The  totality  of 
the  experimental  data  sets  for  evaluat- 
ing a given  model  type  (e.g.,  propaga- 
tion loss  models  with  range  independent 
inputs)  cannot  and  is  not  expected  to  be 
complete,  either  acoustically  or  envi- 
ronmentally. The  sets  are,  rather,  taken 
together,  intended  to  provide  a broad 
range  of  representative  conditions. 
Thus,  for  example,  propagation  loss  data 
sets  should  contain  results  from  many 
ocean  basins,  representing  various  sound 
speed  profile  types  including  some  with 
and  without  depth  excess,  a surface 
duct , those  having  shallow  and  deep 
sound  channels,  subsurface  channels  and 
double  channels.  Bottom  types  and  bottom 
depths  should  be  varied . A range  of  fre- 
quencies covering  surveillance,  tactical 
sonar  and  weapons  sonars  should  be  rep- 
resented, as  should  various  source/ 
receiver  depth  combinations.  Finally, 
the  various  propagation  modes  of  surface 
duct,  bottom  bounce,  and  convergence 
zone  should  be  represented.  Obviously, 
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(U)  Table  3.  Experimental  data  for  use  in  accuracy  assessment 
of  range  independent  propagation  loss  models 
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complete  coverage  of  these  various  pa- 
rameters Implies  an  environmental/ 
acoustic  matrix  of  enormous  size.  Only 
for  a few  elements  of  this  matrix  are 
experimental  data  available.  This  neces- 
sitates the  use  of  environmental  acous- 
tic scenarios  unsupported  with  experi- 
mental propagation  loss  data.  The  envi- 
ronmental acoustic  scenarios  unsupported 
by  experimental  data  are  used  by  com- 
paring results  from  the  model  under 
evaluation  with  those  of  a reference 
model.  These  comparisons  are  used  in  a 
diagnostic  sense  to  identify  model  limi- 
tations and  failures.  (In  this  context, 
both  the  model  under  evaluation  and  the 
reference  model  are  considered  suspect 
when  discrepancies  are  found.)  Under  the 
auspices  of  AMEC,  twelve  sets  of  data 
have  been  identified,  examined,  and 
deemed  suitable  for  range  independent 
propagation  loss  model  evaluation  (l.e., 
satisfied  criteria  of  range  independ- 
ence, contained  sufficient  data  density, 
and  have  adequate  supporting  environmen- 
tal data).  These  sets  are  described  in 
Martin  (1981)  and  are  designated  as 
SUDS,  GULF  OF  ALASKA,  FASOR,  LORAD, 
PA  MCA  IT,  HAYS -MUR  PdY,  JAGUAR-BRASIL, 
BEARING  STAKE,  JOAST,  ATOE,  and  IOMEDEX. 
Table  3 summarizes  some  basic  character- 
istics of  eight  of  these  sets,  which  in- 
dicates that  they  are  representative  of 
the  matrix  giving  the  totality  of  envi- 
ronmental/acoustic scenarios.  We  note 
that  each  experimental  data  set  can  con- 
sist of  many  data  records  corresponding 
to  different  frequencies,  tow  tracks, 
source/receiver  depth  combinations,  etc. 


their  applicability  to  range  dependent 
propagation  loss  model  evaluation  are 
ROUGH  START,  SQUARE  DEAL,  NORLANT  72, 
ATOE,  PARKA  II,  TRANS LANT  I,  NEAT  II, 
S JDS , and  BEARINC  STAKE. 

(U)  It  is  often  desired  to  assess  the 
accuracy  of  a model  for  an  environmen- 
tal scenario  unsupported  by  acoustic 
experimental  data.  In  9uch  cases  the 
model  results,  as  mentioned  above,  are 
to  be  compared  with  the  outputs  of  or.e 
or  more  reference  models.  A reference 
model  Is  not  to  be  considered  a "stand- 
ard." An  attempt  is  made,  however,  to 
choose  a reference  model,  the  physical 
and  mathematical  basis  of  which  19  more 
rigorous  than  that  of  the  model  to  be 
evaluated.  In  comparing  results  from  two 
models  it  Is  usually  essential  that  the 
models  be  run  with  Identical  data  bases 
and  that  the  data  bases  be  separated 
from  the  models.  For  example,  In  two 
propagation  loss  models,  the  bottom  loss 
may  be  a subroutine  of  each  model.  For 
valid  comparison  of  propagation  loss 
results  the  bottom  losses  must  be  ident- 
ical (which  may  require  alteration  of 
one  or  both  models).  Here,  a difficult 
practical  issue  arises.  In  Its  applica- 
tion by  a user,  a model  is  often  adopted 
with  a data  base,  exemplified  by  bottom 
loss.  The  potential  user  would  like  to 
see  the  model/data  base  package  evalua- 
ted. This  Is  accomplished  by  raeanr  ? 
factor  isolation  whereby  for  a given 
scenario  only  a single  factor,  such  as 
bottom  loss,  Is  varied.  The  scenario  is 
chosen  so  that  the  factor  of  Interest  Is 


For  the  evaluation  of  a specific  model, 
a subset  of  the  available  data  records 
would  bo  selected  for  accuracy  assess- 
ment. The  choice  of  this  subset  would  be 
determined  by  the  model’s  Intended  do- 
main of  applicability.  Tables  4-14  give 
synopses  of  important  features  for  each 


dan  inant.  Thus,  the  model  may  be  run 
wLth  infinite  bottom  loss,  a bottom  loss 
table  In  fleet  usage,  bottom  loss  deter- 
mined from  a model  containing  subbottom 
structure,  etc.  The  effect  of  the  factor 
Is  thereby  assessed  for  the  model  being 
evaluated . 


data  set  of  the  pool  for  use  in  range 
independent  propagation  loss  model  eval- 
uation. The  experimental  data  sets  here- 
in described  are  not  to  be  though  of  as 
a final  selection  but  rather  those  ex- 
amined and  summarized  to  date.  Data  sets 
identified  for  examination  to  determine 


(U)  Sometimes  In  comparing  two  models, 
•■he  use  of  identical  data  bases  Is  lm- 
'osslble,  but  some  equivalence  must  be 
achieved.  To  illustrate,  consider  two 
ambient  noise  models  (horizontally 
directional  at  low  frequency).  One,  the 
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Frequency 

(kHz) 


Source 
Depth  (m) 


Rece  i ver 
Depths  (m) 


140 

143 

124 

108 

107 

1 12A 

112B 

1.5 

1.5 

1.5 

2.5 

2.5 

2.5 

2.5 

30.5 

30.5 

30.5 

1067 

1067 

305 

305 

1 5,30.5 
90,229,305 

Same 

Same 

Same 

Same 

Same 

Same 

40.6 

9.5 

3.1 

2.8 

32.9 

16.6 

2.1 

69.3 

58 

12.1 

30.9 

73.5 

63.9 

20.3 

4078 
1 525.4 

4042 
1 524.7 

4042 
1 524.7 

4060 
1 1 .0 

4060 
1 525.0 

4060 
1 525.0 

4042 

1524.7 

10 

1476.7 

10 

1476.8 

10 

1476.5 

0 

1477.6 

10 

1476.6 

10 

1479.1 

10 

1479.2 

1461 .9 
75 

1462.5 

90 

1461.9 

75 

1462.5 

85 

1462.5 

85 

1462.0 

75 

1461 .9 
75 

1-5 

11-15 

31-35 

6-10 

■ ,j 

16-20 

21-25 

26-30 

Max  imum 
Range  (kyd) 


Bottom 
Depth  (m) 

S.  Speed  (m/s) 


Layer 
Depth  (m) 

S.  Speed  (m/s) 


S.  Speed 
Minima  (m/s) 
Depth  (m) 


NAVDAB 
EXP.  9 
RUN  NO. 


VIGATION  - Range  determined  by  clock  differences  times  sound  speed:  Accuracy  100  yds 
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(U)  Table  6.  Parameters  for  FASOP  ‘ntions 


FASOR  Station 


REDWDOO  OAK  THORN  INDIA 


IULIETT 


NAVDAB:  Exp. 

Stat ion 
Run 


Swe II  Height 
(ft) 

6 

Bottom  Depth 
(m) 

7648 

UNCLASSIFIED 
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(U)  Table  8.  PARKA  IIA  parameters 


Comment:  PARKA  Events  9-2  end  9-3  ere  the  only  ones  with  neerly 

renge  Independent  environments  over  the  entire  tracks. 

These  events  resulted  in  neerly  Identicel  propegetlon  toss 
level  vs.  renge  end  therefore  only  Event  9-2  Is  recorrmended 
for  eveluetlon 


Event 


H 


Source  type 

Detonetlon  depth  control 
Source  depths  (ft) 


3 lb.  TNT  blocks 


Fuse  cut  to  length 


60  end  S00 


Receiver  depths  (ft) 


300/2500/10,800 


Anelysis  frequency  (Hz) 
Anelysis  bendwidth/type 
Min  renge  (nm) 

Mex  renge  (nm) 


25/50/100/180/400 
1/3  octeve/totel  energy 


Surfece  sound  speed  (ft/sec) 

Leyer  depth  (ft)/sound  speed  (ft/sec) 


5022.16 


262.5/5026.49 


Sound  ex  I s depth  (ft)/sound  speed  (ft/sec)  3280.8/4857.19 


Bottom  depth  (ft) 


18,600 


Nev I get  I on 


Redio  tone  0.1  nm  eccurecy 


Date  locetlon  - NUSC  dlgitel  tepe  stored  with  others  et  Federal 
Records  Center,  We  I them,  MA.  Rec.  Group  #1 81 , 
Accession  #75-A-342  FRC  Box  #425582  or  425583. 

Reel  with  label  P2RR3  (Box  5 of  Aug  74  shipment  to 
Waltham),  NUSC  po int-of-contact  - Stan  Jackson. 
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(U)  Table  9.  HAYS-AAURPHY  parameters 


Exerci $e: 

Event : 

Source  Type: 

Source  Depths: 
Receiver  Depths: 
Acous t i c Ana  I ys i s : 

Fr equenc i es : 

Data  Density: 

Range  over  which 
Environment  is 
Range  Independent: 

Water  Depth: 

Environmental  Data: 
Nav I gat i on: 

Range  Determination: 
Range  Accuracy: 

Data  Location: 
Comments : 


Mediterranean  Sound  Transmission  (1968) 

ST.  MARGARETS 

1 lb.  TNT 

80  and  325  ft 

350,  450  and  1000  ft 

1/3  octave  total  energy 

35,  67.5,  100  and  200  Hz 

80  ft  source,  3 shots  per  nm 
350  ft  source,  1 shot  per  10  nm 


0-240  nm 

1500  fathoms  max  (2750  m) 

1420  fathoms  min  (2600  m) 

Sound  velocity  profiles  and  water  depth  vs.  range 
Not  addressed  in  report  as  such 
Radio  link  time  difference 

Less  than  1.5%  of  range  — probably  better  than 
0.5%  of  range 

Each  plotted  value  tabulated  vs.  range  in  Reference  (1) 

(1 ) Source  level  accuracy  is  3 dB  or  better 

(2)  Depth  excess  is  1500  m or  more.  Bottom  reflected 
energy  may  dominate  below  67.5  Hz  but  probably 
contribute  little  above  100  Hz. 
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(U)  Table  10,  JAGUAR  exercise  parameters  summary 
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(continued).  Parameters  for  Bearing  Stake  exercise,  Site  IB,  VAC 
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(U)  Table  1 1 (continued).  BEARING  STAKE  parameters,  Site  4,  VAC 


Bottom 

Depth 

(m) 


Bottom  depth  is  approximately  5105  m to  about  295  km 
along  the  track. 
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(U)  Table  13.  ATOE  parameter  depressed  sound  channel  measurement 


• 02 5 1 .05, .1 , 

• 2, .4, .8,1 .8 


♦Average  Value  min  4390,  max  5487.  Since  the  source  Is  omnidirectional  and  the 
received  signal  is  integrated  over  all  paths,  bottom  effects  are  initially 
effective  in  range.  (See  also,  factors  which  may  effect  model  comparison.) 

UNQASSFIED 


(U)  Table  14.  Parameters  for  the  IQMEDEX  data  set 


Figure  6 


Frequency  (Hz) 

125 

125 

125 

125 

125 

125 

Source  Depth  (m) 

152 

152 

152 

152 

152 

152 

Receiver  Depth  (m) 

137 

613 

1113 

1116 

2377 

2650 

Mn  Range  (nm) 

8 

8 

8 

8 

8 

8 

Max  Range  (nm) 

1 2C 

135 

138 

135 

120 

120 

Layer  Depth  (m) 

42 

42 

42 

42 

42 

42 

Sound  Axi s (m) 

The  axis  of  min 

sound  speed  is  about  130 

m 

Critical  Depth 

The  mean  critical  depth 

is  1000  i 

T) 

Bottom  Depth 

Assumed 

constant 

t at  3000 

1 m 

Navi  gat  ion 

Range  accuracy  v 

vas  + 0.2 

nm  (see 

Reference  2) 

Data  Location 


Data  can  be  accessed  through  the  "LRAPP  Acoustic 
Data  Bank"  by  request  to  NORDA  Code  520  SEAS 
Project,  NSTL  Station,  MS  39529 
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model  to  be  evaluated,  uses  a statisti- 
cal ship  distribution.  The  other,  the 
reference  model,  is  more  detailed  and 
represents  the  ships  as  point  sources. 
In  comparing  these  models,  it  is  essen- 
tial that  the  point  model  have  ship  sta- 
tistics equivalent  to  those  of  the  model 
which  uses  a statistical  ship  distribu- 
tion. 

(U)  The  use  of  a reference  model  is  in- 
dicated in  the  following  situations:  (1) 
no  experimental  data  is  available  for  a 
scenario  for  which  the  model  is  intend- 
ed, (2)  comparison  of  the  model  under 
evaluation  with  experimental  data  has 
resulted  in  unexplainable  discrepancies, 
(3)  a specific  feature  of  the  model  re- 
quires evaluation  as  indicated  by  an  ex- 
amination of  the  physics  and  mathematics 
of  the  model,  and  (4)  the  sensitivity  of 
the  model  to  a given  factor  (e.g.,  bot- 
tom loss)  is  to  be  determined.  For  range 
Independent  propagation  loss  models,  a 
set  of  environmental  scenarios  Is  con- 
tained in  Hammond  and  Rublsch  (1976).  A 
map  giving  the  geographic  coverage  rep- 
resented by  the  Hammond  and  Rublsch 
scenarios  is  given  in  Figure  9.  A typi- 
cal set  of  parameters  is  given  in  Table 
13.  These  encompass  bottom  loss,  bottom 
parameters  and  sound  speed  profile. 

(U)  Closed  form  solutions  are  Important 
benchmarks  that  must  be  achieved  by 
models.  A model’s  inability  to  match 
these  solutions  is  an  Indicator  of  er- 
rors in  computer  coding,  overly  simpli- 
fying assumptions  or  approximations,  or 
improper  selection  of  parameters. 

7.0  (U)  Summary 

(U)  The  Acoustic  Model  Evaluation  Com- 
mittee has  developed  a methodology  for 
the  evaluation  of  propagation  loss 
models  using  both  range  independent  and 
range  dependent  environmental  inputs. 
Most,  If  not  all,  of  the  methodology  is 
applicable  to  ambient  noise  and  reverb- 
eration models.  For  propagation  loss 
models,  environmental  acoustic  data  sets 
have  been  reviewed  for  their  appropri- 
ateness to  model  evaluation  from  the 


viewpoint  of  quality,  completeness  of 
environmental  data  for  model  input,  fac- 
tor isolation  (e.g.,  convergence  zone 
propagation),  geographic  coverage,  and 
frequency  coverage.  The  salient  features 
of  these  data  sets  have  been  summarized 
and  reported  for  range  independent  prop- 
agation loss  (Martin,  1981)  and  for 
range  dependent  propagation  loss 
(Martin,  in  prep.).  Subsets  of  the  range 
independent  sets  have  been  acquired  on 
digital  tape,  examined  for  obviously  bad 
data  points,  edited  and,  finally,  put  on 
a tape  that  constitutes  one  part  of  a 
portable  test  package. 

(U)  The  remaining  two  parts  of  the  port- 
able test  package  are  a Model  Informa- 
tion Form  for  the  model  developer  to 
complete  and  a computer  program,  MCHtO, 
for  the  quantitative  comparison  of  two 
data  sets  (Sussraan  and  Oberlander, 
1979).  The  Model  Information  Form  is 
applicable  to  both  range  Independent  and 
range  dependent  propagation  loss  models 
and,  with  minor  additions  and  alter- 
ations, can  be  used  for  ambient  noise 
and  reverberation  model  evaluation.  With 
the  completion  of  the  portable  test 
package  for  range  Independent  propaga- 
tion loss  models,  these  models  can  be 
evaluated  in  10-12  months.  This  does 
not,  however,  mean  that  this  aspect  of 
model  evaluation  is  either  perfect  or 
complete.  The  methodology  has  been  and 
remains  in  an  evolutionary  state.  Im- 
provenents  in  the  methodology  are  indi- 
cated in  at  least  two  areas: 

• (U)  Model  Sensitivity.  To  what  degree 
is  the  output  affected  by  a change  in 
input  over  a realistic  span  of  values? 
For  example,  as  bottom  loss  type  is 
changed  from  1 to  9,  does  the  output 
show  a monotonic  change  of  values  and  is 
there  also  a change  in  the  basic  nature 
of  the  output,  such  as  the  emergence  of 
convergence  zones? 

• (U)  Fluctuations.  The  accuracy  assess- 
ment procedure  used  by  AMEC  is  based  on 
mean  levels  (i.e.,  the  fluctuating  com- 
ponent is  removed).  However,  the  fluctu- 
ating component  of  propagation  loss  is 
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(U)  Table  15.  Area  X Noras 


(C)  1.  Location:  244N,  63*W 

(U)  2.  Submarine  Province:  Nares  Abyssal  Plain 

(U)  3.  Water  Depth:  About  3100  fathoms  (5890  m)  throughout  the  area. 

(U)  4.  Sediment  Type:  Pelagic  red  clay  with  layers  of  acoustic  reflecting 
material,  typically  limestone  and  cherts. 


(U)  5.  Sediment  Thickness:  200  - 300  m. 

(U)  6.  Bottom  Loss:  Low  to  intermediate 

(U)  7.  Sound  Speed  Profile: 


Depth  Excess 

M.D 

IWD 

Winter 

1600  m 

50  m 

1370  m 

Sumner 

1240  m 

30  m 

1370  m 

(U)  8.  Ship  Densl ty: 


a.  Total  Ships/104  sq  * 77 

b.  Shlps/sq  mile  * 2.36  X 10-4 

c.  Percent  Fishing  Vessels  a 15% 


(U)  9.  Ambient  Noise  level  (uPa)  as  a function  of  frequency  and  depth: 


50 

100 

160 

315 

500 

630 

1000 

90  ft  (27.4  m) 

86.0 

78.5 

73.1 

66.4 

65.3 

64.2 

61.9 

300  ft  (91  .4  m) 

85.2 

80.5 

74.9 

68.7 

61 .3 

54.6 

47.6 

1000  ft  (304.8  m) 

79.1 

73.1 

68.8 

62.7 

62.4 

60.9 

58.6 

The  levels  at  500,  630,  and  1000  Hz  vary  with  the  conditions  of  the  seas. 
The  levels  were  measured  at  300  ft  in  a sea  state  1 and  the  90  ft  and 
1000  ft  In  a sea  state  2. 
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(U)  Table  15  (continued).  Area  X (24°  N,  631  W). 

Bottom  Loss  (dB)  as  a Function  of  Grazing  Angle  and  Frequency 


GRAZING  ANGLE 
(DEGREES ) 

50 

100 

160 

315 

600 

630 

1000 

0 

Ull 

KB 

■ 

5 

0 

.5 

1.0 

1.5 

2 

a 
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0 

1 534.1 

0 

1 544.2 

46 

1 535.0 

30 

1544.8 

76 

1 533.4 

50 

1 542.7 

244 

1521.* 

75 

1538.6 

427 

1 519.9 

100 

1532.8 

549 

1 514.8 

125 

1534.0 

792 

1 500.2 

130 

1531.9 

975 

1494.1 

200 

1527.7 

1097 

1492.9 

250 

1 525.0 

1372 

1492.3 

300 

1 523.2 

1676 

1495.0 

400 

1521.6 

2438 

1504.5 

500 

1517.4 

3048 

1513.0 

549 

1 514.8 

4267 

1532.2 

792 

1500.2 

5486 

1 554.2 

975 

1494.1 

5890 

1561.4 

1097 

1492.9 

1377 

1492.3 

1676 

1495.0 

2438 

1 504.5 

3048 

1513.0 

4267 

1 532.2 

5486 

1554.2 

5890 

1561.4 

CONFIDENTIAL 


also  important  for  the  prediction  of  so- 
nar system  performance.  Qoe  typical 
fluctuation  measure  is  the  standard  de- 
viation of  the  signal  excess  which  is 
partially  based  on  the  standard  devia- 
tion of  the  propagation  loss.  Of  still 
more  basic  interest  is  the  probability 
density  fun-. ton  of  the  transmission 
loss  (Urick,  1975). 

(U)  With  the  additions  listed  above, 
AMEC  has  essentially  developed  an  ap- 
proach for  propagation  loss  model  evalu- 
ation with  the  following  characteris- 
tics: 

• (U)  Objective.  A methodology  has  been 
developed  which  is  applied  to  any  model 
in  a straightforward  manner.  Of  partic- 
ular note  is  the  quantitative  aspect  of 
accuracy  assessment.  A recognized  expert 
reviews  and  reports  upon  the  physics  and 
mathematical  basis  of  the  model. 

• (U)  Standardized.  The  same  Model  In- 
formation Form  is  to  be  utilized  for  all 
models  and  filled  out  by  the  model  de- 
veloper.* The  same  data  bank  is  avail- 
able for  model/measurement  comparison 
for  any  model  to  be  evaluated  making 
comparisons  between  models  relatively 
easy.  Even  plot  formats  are  standard- 
ized to  facilitate  model-to-model  com- 
parisons . 

• (U)  Rapid.  Given  the  portable  test 
package  consisting  of  the  Model  Informa- 
tion Form,  a computer  program  for  quan- 
titatively comparing  two  data  sets 
(usually  model  output  with  measured 
values),  and  a set  of  environmental/ 
acoustic  scenarios  (available  on  digital 
form),  the  evaluation  of  a model  will 


*The  first  two  models  evaluated,  FACT 
PL9D  and  RAYMODE  X,  were  not  put  through 
fhis  process,  since  it  was  being  devel- 
oped during  those  models'  evaluations. 
The  information  requested  in  the  Model 
Information  Form  was,  however,  collected 
by  the  evaluators  with  the  cooperation 
of  the  RAYMODE  model  developer  and  NORDA 
Code  320,  which  is  responsible  for  FACT 
configuration  management. 


now  require  approximately  10-12  months. 
This  time  should  decrease  with  further 
experience . 

(U)  The  benefits  derived  from  evaluating 
models  are  threefold: 

• (U)  The  limitations  of  a model  and, 
hence,  its  domain  of  applicability  are 
identified . The  limitations  may,  in 
their  basic  form,  arise  from  assumptions 
or  approximations  involved  in  the  phys- 
ics or  mathematics,  the  computer  imple- 
mentation or  characteristics  of  the  com- 
puter, telemetry  or  graphics  systems. 
These  translate  Into  limitations  in  fre- 
quency, range  (for  propagation  loss 
models),  or  environment  (e.g.,  flat 
ocean  bottom).  For  example,  run  time  or 
core  limitations  may  reflect  in  frequen- 
cy limitations  for  a normal  mode  propa- 
gation loss  program.  The  small  angle 
approximation  inherent  in  the  parabolic 
equation  method  of  propagation  loss  pre- 
diction may  result  in  invalid  results  at 
ranges  where  large  bottom  slopes  are  en- 
countered. Knowing  the  domain  of  appli- 
cability of  models  is  particularly  im- 
portant for  those  models  in  extensive 
fleet  use,  where  a wrong  answer  may  be 
worse  than  no  answer.  Such  knowledge 
also  indicates  the  need  for  other  models 
and  improvements. 

• (U)  Models  may  be  Improved  through 
Identification  of  model  deficiencies  and 
errors . In  the  course  of  model  evalua- 
tions, errors  in  computer  coding  may  be 
found.  Even  more  likely  is  the  discovery 
of  model  deficiencies,  either  by  design 
(i.e.,  neglected  features  such  as  source 
or  receiver  beam  pattern  capability)  or 
due  to  not  choosing  the  best  technique 
available  for  a given  task.  One  model 
evaluation  responsibility  is  to  recom- 
mend upgrades  or  corrections  to  evalu- 
ated models.  This  function  is  extremely 
important,  since  it  leads  to  Improved 
versions  of  models  with  broader  capabil- 
ities or  increased  accuracy  or  decreased 
core  or  run  time  requirements. 
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• (U)  Mode"  ? may  be  compared  and, 
neace,  the  best  selected  for  a given  ap- 
plication. Given  a number  of  models 
which  have  been  evaluated  through  a 
standardized  evaluation  process,  direct 
model- to-mod el  comparison  is  rather  easy 
and  with  the  addition  of  information  re- 
lating to  specific  aspects  of  the  appli- 
cation not  included  in  the  AMEC  evalu- 
ation, should  lead  to  the  selection  of 
the  best  model . 
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